As described in U.S. Pat. No. 5,347,306, issued to Tohei Nitta on Sep. 13, 1994, and incorporated herein by reference, an electronic meeting place can be provided in which conferees or participants can meet through control of on-screen characters, the movements of which are under control of the individuals. The system described in this patent utilizes cartoon characters or icons in order to avoid the problems with full-frame video teleconferencing. However, in the portrayal of one's self in terms of an on-screen character, it is necessary to provide this character with a persona, style, or mood specified by the individual participant. As described in this patent, the persona is controlled by switches on a keyboard or in a separate box so that full control can be provided over the representation of the individual participant.
It will be appreciated that the utilization of separate switches and key strokes is a cumbersome way to control the persona of the on-screen character. While it might be possible to use the traditional computer mouse to in some way control character persona or style, such a system is not easy to use and requires the individual to be at the computer terminal. It would not be possible when using the computer mouse to sit-back in one's chair or move around during teleconferencing. There is therefore a need for a more practical and simpler way to control the on-screen characters.
More particularly, as is well known, a mouse is utilized to control cursors represented on a computer screen in which the position of the cursor is controlled through the frictional contact of the mouse's roller balls with a fixed surface, usually in the form of a so-called mouse pad. While the utilization of a mouse for computer control is indeed an efficient method for control of computer operations, the mouse has not been utilized to control the persona or style of icons or on-screen characters but rather has been used to move on-screen objects by clicking on them and dragging them to different locations, with the length of the move proportional to the movement of the mouse over the mouse pad. The traditional mouse is therefore incapable of controlling the persona of a cartoon or other character which is presented on-screen in a conferencing or meeting scenario to represent the individual communicating with others in an electronic meeting place or other environment.
Specifically, in order to get a graphical object on a screen to move in a given direction with a mouse, one must click on the object and then move the mouse to drag it by specifying each location along the desired path. Thus a graphical object cannot be made to move in a desired direction without specifying each point along the way. As a result, simple movements of the graphical object cannot be made by merely directing the graphical object to go in a desired direction. Thus the mouse is inefficient both to specify simple movements and to give a graphical object a desired persona.
By way of further background as indicated in the above-mentioned patent, the movement of the individual characters representing a particular person can be controlled through data gloves, data suits, chair input devices and the like. These devices are in some sense cumbersome, and it is therefore desirable to provide a convenient and preferably hand-held device which controls not only the movement of one's individual character, but also the mood or the persona which the individual wishes to impart to his on-screen character.
As mentioned above, in this patent cartoon characters are prestored to dramatically reduce the bandwidth required for teleconferencing. During the teleconferencing scenario, the individuals participating in a conference need to be able to control how they are presented to the conferees. This is accomplished through the provision of the associated persona commands, in one embodiment by providing physical switches at each of the terminals for each of the participants in a distributed teleconferencing network.
However, the utilization of individual switches, or even a mouse driving a cursor on a screen menu is a cumbersome way in which to control what other conferees see of the individual conferee. There is therefore a requirement for a hand-held device which can be free of contact with a table or other flat surface that can be conveniently waved around by an individual conferee to permit control of his character without the requirement of wearing a data suit or a data glove, without the requirement of detecting hand or body position through optical recognition techniques which are computer intensive and slow, and without any physical attachment to a fixed structure to sense hand or finger position.
Again, by way of background, hand gestures have been utilized in communication systems for the deaf. U.S. Pat. No. 5,047,952 describes an instrumented glove to produce signals corresponding to hand configuration utilizing strain sensors. A system for converting an image to an animated cartoon-like drawing through a so-called contour extractor is illustrated in U.S. Pat. No. 4,546,383. U.S. Pat. No. 4,884,972 utilizes a mouse to select words for animated characters, whereas U.S. Pat. No. 4,906,940 describes a system for extracting features and images for machine control. Moreover, U.S. Pat. No. 5,008,946 describes a system sensing the movement of one's pupil and one's mouth to control a car, whereas U.S. Pat. No. 4,414,537 shows a man-machine interface utilizing a data glove. U.S. Pat. No. 4,567,610 illustrates a reference histogram for pattern recognition, whereas the following articles and thesis describe hand gesture recognition systems and adaptive interfaces relating to hand gestures:
Tomoichi TakahaShi and Fumio Kishino, Hand Gesture Coding Based on Experiments Using a Hand Gesture Interface Device, SICCHI Bulletin, April 1991; Interpreting Sign Language, IEEE Computer Graphics & Applications, January 1994, pps. 36-37; Jennifer A. Hall, The Human Interface in Three-Dimensional Computer Art Space, media Lab, Massachusetts Institute of Technology, MSVS Thesis, Oct. 18, 1985; and, S. Sidney Fels, Building Adaptive Interfaces with Neural Networks: The Glove-Talk Pilot Study, Department of Computer Science, University of Toronto, Tech Report No. CRG-TR-90-1, February 1990.
It will be seen that while there are various methods of hand gesture recognition, they rely on multiple hand gestures for computer control. As such, signing recognition is much too complicated for machine control, as the difference in the various hand signs are difficult to recognize, and the language associated with hand signing too complex for machine control.
Note that in terms of arcade games, joy sticks, and the like control graphical objects through the use of strain gages and switches, with the strain being measured between a fixed structure and the joy stick. These are simple systems which do not analyze strain gage waveforms nor combine waveforms for graphical object control. Thus, there are no comparisons or correlations with templates that can be used to establish the existence of a given gesture or series of gestures. As a result television game control devices generally do not provide a sufficient level of sophistication to control the persona of graphical objects representing human beings.
Furthermore, such systems do not provide for adjustments in the attributes of character motion or control. When a motion is performed, it is always performed in exactly the same manner. On the other hand, persons have different characteristics to their motions depending upon the person, the emotional state of the person, or context of the motion. All of the gesturing systems fail to provide for variable characteristics of personal motion.
Interactive programs permit the control of certain characters in relation to other characters provided by a program. Such interactive programs may include theatrical productions or virtual reality systems. For such systems, the total control of the character, including variation in the attributes of motion, would provide a more realistic representation of a character. Existing systems do not permit such operations. Furthermore, with interactive programs, speech synthesis provides a portion of the interaction. Variation of the voice quality is imperative to provide a realistic character as situations change. Existing control systems do not permit variations in speech synthesis.